Neural Architecture Transfer

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete each deployment specification of hardware or objective. This is computationally impractical endeavor given the potentially large number application scenarios. In this paper, we propose Architecture Transfer (NAT) to overcome limitation. NAT designed efficiently generate custom models that are competitive under multiple conflicting objectives. To realize goal learn supernets from which specialized subnets can be sampled without any additional training. The key our approach an integrated online transfer learning and many-objective evolutionary procedure. A pre-trained supernet iteratively adapted while simultaneously searching subnets. We demonstrate efficacy on 11 benchmark image classification tasks ranging large-scale multi-class small-scale fine-grained datasets. all cases, including ImageNet, NATNets improve upon state-of-the-art mobile settings ($\leq$ 600M Multiply-Adds). Surprisingly, datasets benefit most NAT. At same time, orders magnitude more efficient than existing methods. Overall, experimental evaluation indicates that, across diverse computational objectives, appreciably effective alternative conventional fine-tuning weights network learned standard Code available at https://github.com/human-analysis/neural-architecture-transfer